Methods and systems for collaborative control of a remote vehicle

ABSTRACT

Systems and methods are for enabling a group of individuals, each using an individual computing device, to collaboratively control a vehicle in real-time as a unified intelligence. The collaboration system comprises a plurality of computing devices, each of the devices being used by an individual user, each of the computing devices enabling its user to contribute to the emerging real-time group-wise intent. A collaboration server is disclosed that moderates the closed-loop system, enabling convergence upon a unified group intent. Control commands are repeatedly sent to the vehicle based on the determined group intent.

This application claims the benefit of U.S. Provisional Application No.62/463,657 entitled METHODS AND SYSTEMS FOR COLLABORATIVE CONTROL OF AROBOTIC MOBILE FIRST-PERSON STREAMING CAMERA SOURCE, filed Feb. 26,2017, which is incorporated in its entirety herein by reference.

This application claims the benefit of U.S. Provisional Application No.62/473,429 entitled METHODS AND SYSTEMS FOR COLLABORATIVE CONTROL OF AROBOTIC MOBILE FIRST-PERSON STREAMING CAMERA SOURCE, filed Mar. 19,2017, which is incorporated in its entirety herein by reference.

This application is a continuation-in-part of U.S. application Ser. No.14/668,970 entitled METHODS AND SYSTEMS FOR REAL-TIME CLOSED-LOOPCOLLABORATIVE INTELLIGENCE, filed Mar. 25, 2015, which in turn claimsthe benefit of U.S. Provisional Application 61/970,885 entitled METHODAND SYSTEM FOR ENABLING A GROUPWISE COLLABORATIVE CONSCIOUSNESS, filedMar. 26, 2014, both of which are incorporated in their entirety hereinby reference.

This application is a continuation-in-part of U.S. application Ser. No.14/708,038 entitled MULTI-GROUP METHODS AND SYSTEM FOR REAL-TIMEMULTI-TIER COLLABORATIVE INTELLIGENCE, filed May 8, 2015, which in turnclaims the benefit of U.S. Provisional Application 61/991,505 entitledMETHODS AND SYSTEM FOR MULTI-TIER COLLABORATIVE INTELLIGENCE, filed May10, 2014, both of which are incorporated in their entirety herein byreference.

This application is a continuation-in-part of U.S. application Ser. No.14/738,768 entitled INTUITIVE INTERFACES FOR REAL-TIME COLLABORATIVEINTELLIGENCE, filed Jun. 12, 2015, which in turn claims the benefit ofU.S. Provisional Application 62/012,403 entitled INTUITIVE INTERFACE FORREAL-TIME COLLABORATIVE CONTROL, filed Jun. 15, 2014, both of which areincorporated in their entirety herein by reference.

This application is a continuation-in-part of U.S. application Ser. No.14/859,035 entitled SYSTEMS AND METHODS FOR ASSESSMENT AND OPTIMIZATIONOF REAL-TIME COLLABORATIVE INTELLIGENCE SYSTEMS, filed Sep. 18, 2015which in turn claims the benefit of U.S. Provisional Application No.62/066,718 entitled SYSTEM AND METHOD FOR MODERATING AND OPTIMIZINGREAL-TIME SWARM INTELLIGENCES, filed Oct. 21, 2014, both of which areincorporated in their entirety herein by reference.

This application is a continuation-in-part of U.S. application Ser. No.14/920,819 entitled SUGGESTION AND BACKGROUND MODES FOR REAL-TIMECOLLABORATIVE INTELLIGENCE SYSTEMS, filed Oct. 22, 2015 which in turnclaims the benefit of U.S. Provisional Application No. 62/067,505entitled SYSTEM AND METHODS FOR MODERATING REAL-TIME COLLABORATIVEDECISIONS OVER A DISTRIBUTED NETWORKS, filed Oct. 23, 2014, both ofwhich are incorporated in their entirety herein by reference.

This application is a continuation-in-part of U.S. application Ser. No.14/925,837 entitled MULTI-PHASE MULTI-GROUP SELECTION METHODS FORREAL-TIME COLLABORATIVE INTELLIGENCE SYSTEMS, filed Oct. 28, 2015 whichin turn claims the benefit of U.S. Provisional Application No.62/069,360 entitled SYSTEMS AND METHODS FOR ENABLING AND MODERATING AMASSIVELY-PARALLEL REAL-TIME SYNCHRONOUS COLLABORATIVESUPER-INTELLIGENCE, filed Oct. 28, 2014, both of which are incorporatedin their entirety herein by reference.

This application is a continuation-in-part of U.S. application Ser. No.15/017,424 entitled ITERATIVE SUGGESTION MODES FOR REAL-TIMECOLLABORATIVE INTELLIGENCE SYSTEMS, filed Feb. 5, 2016 which in turnclaims the benefit of U.S. Provisional Application No. 62/113,393entitled SYSTEMS AND METHODS FOR ENABLING SYNCHRONOUS COLLABORATIVECREATIVITY AND DECISION MAKING, filed Feb. 7, 2015, both of which areincorporated in their entirety herein by reference.

This application is a continuation-in-part of U.S. application Ser. No.15/047,522 entitled SYSTEMS AND METHODS FOR COLLABORATIVE SYNCHRONOUSIMAGE SELECTION, Feb. 18, 2016 which in turn claims the benefit of U.S.Provisional Application No. 62/117,808 entitled SYSTEM ND METHODS FORCOLLABORATIVE SYNCHRONOUS IMAGE SELECTION, filed Feb. 18, 2015, both ofwhich are incorporated in their entirety herein by reference.

This application is a continuation-in-part of U.S. application Ser. No.15/052,876 entitled DYNAMIC SYSTEMS FOR OPTIMIZATION OF REAL-TIMECOLLABORATIVE INTELLIGENCE, filed Feb. 25, 2016 which in turn claims thebenefit of U.S. Provisional Application No. 62/120,618 entitledAPPLICATION OF DYNAMIC RESTORING FORCES TO OPTIMIZE GROUP INTELLIGENCEIN REAL-TIME SOCIAL SWARMS, filed Feb. 25, 2015, both of which areincorporated in their entirety herein by reference.

This application is a continuation-in-part of U.S. application Ser. No.15/086,034 entitled SYSTEM AND METHOD FOR MODERATING REAL-TIMECLOSED-LOOP COLLABORATIVE DECISIONS ON MOBILE DEVICES, filed Mar. 30,2016 which in turn claims the benefit of U.S. Provisional ApplicationNo. 62/140,032 entitled SYSTEM AND METHOD FOR MODERATING A REAL-TIMECLOSED-LOOP COLLABORATIVE APPROVAL FROM A GROUP OF MOBILE USERS filedMar. 30, 2015, both of which are incorporated in their entirety hereinby reference.

This application is a continuation-in-part of U.S. application Ser. No.15/199,990 entitled METHODS AND SYSTEMS FOR ENABLING A CREDIT ECONOMY INA REAL-TIME COLLABORATIVE INTELLIGENCE, filed Jul. 1, 2016, which inturn claims the benefit of U.S. Provisional Application No. 62/187,470entitled METHODS AND SYSTEMS FOR ENABLING A CREDIT ECONOMY IN AREAL-TIME SYNCHRONOUS COLLABORATIVE SYSTEM filed Jul. 1, 2015, both ofwhich are incorporated in their entirety herein by reference.

This application is a continuation-in-part of U.S. application Ser. No.15/241,340 entitled METHODS FOR ANALYZING DECISIONS MADE BY REAL-TIMEINTELLIGENCE SYSTEMS, filed Aug. 19, 2016, which in turn claims thebenefit of U.S. Provisional Application No. 62/207,234 entitled METHODSFOR ANALYZING THE DECISIONS MADE BY REAL-TIME COLLECTIVE INTELLIGENCESYSTEMS filed Aug. 19, 2015, both of which are incorporated in theirentirety herein by reference.

This application is a continuation-in-part of U.S. application Ser. No.15/640,145 entitled METHODS AND SYSTEMS FOR MODIFYING USER INFLUENCEDURING A COLLABORATIVE SESSION OF REAL-TIME COLLABORATIVE INTELLIGENCESYSTEM, filed Jun. 30, 2017, which in turn claims the benefit of U.S.Provisional Application No. 62/358,026 entitled METHODS AND SYSTEMS FORAMPLIFYING THE INTELLIGENCE OF A HUMAN-BASED ARTIFICIAL SWARMINTELLIGENCE filed Jul. 3, 2016, both of which are incorporated in theirentirety herein by reference.

This application is a continuation-in-part of U.S. application Ser. No.15/815,579 entitled SYSTEMS AND METHODS FOR HYBRID SWARM INTELLIGENCE,filed Nov. 16, 2017, which in turn claims the benefit of U.S.Provisional Application No. 62/423,402 entitled SYSTEM AND METHOD FORHYBRID SWARM INTELLIGENCE filed Nov. 17, 2016, both of which areincorporated in their entirety herein by reference.

This application is a continuation-in-part of U.S. application Ser. No.15/898,468 entitled ADAPTIVE CONFIDENCE CALIBRATION FOR REAL-TIME SWARMINTELLIGENCE SYSTEMS, filed Feb. 17, 2018, which in turn claims thebenefit of U.S. Provisional Application No. 62/460,861 entitledARTIFICIAL SWARM INTELLIGENCE WITH ADAPTIVE CONFIDENCE CALIBRATION,filed Feb. 19, 2017 and also claims the benefit of U.S. ProvisionalApplication No. 62/473,442 entitled ARTIFICIAL SWARM INTELLIGENCE WITHADAPTIVE CONFIDENCE CALIBRATION, filed Mar. 19, 2017, all of which areincorporated in their entirety herein by reference.

This application is a continuation-in-part of International ApplicationNo. PCT/US15/22594, filed Mar. 25, 2015.

This application is a continuation-in-part of International ApplicationNo. PCT/US15/35694, filed Jun. 12, 2015.

This application is a continuation-in-part of international ApplicationNo. PCT/US15/56394, filed Oct. 20, 2015.

This application is a continuation-in-part of International ApplicationNo. PCT/US16/40600, filed Jul. 1, 2016.

This application is a continuation-in-part of International ApplicationNo. PCT/US17/40480, filed Jun. 30, 2017.

This application is a continuation-in-part of International ApplicationNo. PCT/US2017/062095, filed Nov. 16, 2017.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates generally to systems and methods forenabling a real-time, unified, collective intelligence among networkedusers, and more specifically, to collaborative control systems andmethods for vehicle control.

2. Discussion of the Related Art

Over recent years, live internet streaming services like PERISCOPE®,FACEBOOK® Live, and YOUTUBE® Live, have become extremely popular,enabling individual users to stream real-time video content to largegroups of followers. In some instances, this real-time video is “firstperson” video, captured from the perspective of a (a) a single humanuser, (b) a remotely controlled unmanned aerial vehicle (e.g. a drone),or could be provided from (c) a robotically controlled (i.e. autonomous)unmanned aerial vehicle or other robotic camera source. Regardless ofthe camera source, this streaming model for live content is oftenreferred to as a “one to many” architecture because the contentoriginates from a camera source associated with a one user (or onedrone) and terminates at a large collection of separate users (i.e.many) who all view the content at substantially the same time. Thesource content can be streamed to the internet (via a streaming service)from a mobile computing device on the person of the source user orsource robot, allowing a large number of followers to experience thelive content as real-time voyeurs of the source user, having a vicariousexperience of being in the place and time of the source users. This issometimes referred to as “Telepresence” or “First Person Video” (FPV),especially if the video content captured from the first-personperspective of the source user or robot and then presented asfirst-person content to the receiving user, for example through videoheadsets or fixed position immersive video displays. In this way, alarge number of receiving users can be given the perspective of being atthe location of the source user or source drone, thus having a vicariousexperience of that location as the source user or source drone navigatesthe physical space.

In one example of FPV, a camera source is associated with the sourceuser or source aerial vehicle. The source can be held by a person, butfor applications that stream first person video (FPV), the camera sourcecould also be head-worn, for example using a camera-enabled headset. Insome embodiments, the camera source could be a stereo camera pair, or 3Dcamera system, along with stereo microphones. This enables the sourceuser to stream immersive video and audio content that provides highlyrealistic experiences for receiving users. In some examples, the camerasource can be mounted upon a remote-controlled flying drone or a roboticdrone, providing the perspective to viewing users of flying along withor driving along with the drone. As commonly known in the art, aremote-controlled drone system can include a drone and a pan-and-tiltcamera coupled to the drone. The drone system includes a localcontroller which is configured to receive commands over wirelesscommunication from a remote device (such as a remote controller) andstream video footage (and optionally audio) to the remote device.

The commands received by the drone are generally high-level instructionsabout guidance (e.g. drone direction, drone speed, and optionally dronealtitude) as well as camera control (e.g. pan, tilt, and zoom), whichare interpreted by the drone local controller, which modulates thepropeller actuators and camera actuators accordingly. In this way, ahuman user who is watching the video feed at the remote device (or otherdevice in communication with the remote device), and manipulating a userinterface in communication with the remote device, can fly (or drive)the drone remotely. A typical remote controller includes a video displayand user interface. In one example, the controller includes a real-timedisplay of streamed video from the drone, as well as joysticks and/orother manually engaged elements for guiding the direction, orientation,altitude, and speed of the drone as well as the pan, tilt, and zoom ofthe camera.

while it's currently common to share the live video with a large numberof online users who can experience the live video from a first-personperspective and thereby have a vicarious experience of being at thecamera location, as known in the art the large number of online usersare passive viewers. In the case of live video from a drone, the passiveviews can experience the content from the perspective of the camera onthe drone, but the guidance of the drone as generally enabled is for asingle user to control. This makes one person the “driver” (or “pilot”)and all other users passive viewers who are merely along for the ride.This greatly reduces the potential engagement for the audience, as theylack the ability to guide the drone in real-time.

When viewing real-time first-person video (FPV), the receiving users canview the video on an immersive headset, with optional stereoscopicimagery and surround sound. Such configurations can be highlyentertaining because it allows the receiving users to experience whatit's like for the source user to navigate the real world from theperspective of the drone. But again, without the ability to influencecontrol the motion of the drone, the user is a passive viewer having animmersive experience rather than an active participant who iscontributing to a real-time intelligence that is controlling the drone.

What is needed is a real-time control system that empowers a group ofnetworked users to work together as a closed-loop system to control themotion of the drone, and optionally control the camera's pan and tiltand zoom. This would provide a very unique and powerful experience inwhich a group of individuals can work together as a unified intelligentsystem (i.e. an Artificial Swarm Intelligence) to guide a drone aroundthroughout a physical environment, tapping their collective will,wisdom, and intelligence. This would also greatly increase engagementand interactivity of the users. It also enables drones to be controlledwith amplified intelligence resulting from the use of human swarms.

To solve these problems, innovative systems and methods are needed toenable groups of users who are simultaneously viewing a real-time videostream from a robotic drone, to work together as a closed-loop system tocontrol the motion of the drone as a unified collective intelligence.

SUMMARY OF THE INVENTION

Several embodiments of the invention advantageously address the needsabove as well as other needs by providing a real-time collaborativevehicle control system comprising: a plurality of computing devices eachcomprising a communications infrastructure coupled to each of aprocessor, a memory, and a collaborative intent application stored oneach memory and configured to run on each processor, and a userinterface coupled to a display and configured to receive input from auser; a collaborative server in networked communication with theplurality of computing devices, the collaboration server including aprocessor and a memory; a vehicle controller configured to receivecommands from the collaborative server; and a vehicle operativelycontrolled by the vehicle controller, wherein the control system isconfigured to perform the following steps, wherein computing devicesteps are performed substantially simultaneously on each of theplurality of computing devices: display, on each of the computingdevices, vehicle control information, and a collaboratively controlledgraphical indication having a coordinate location in relation to thevehicle control information; receive, on each of the computing devices,repeatedly in real-time, user input of a user intent vector through theuser interface, the user intent vector having a direction in relation tothe user control information and a magnitude, send, repeatedly inreal-time, the user intent vector to the collaboration server,responsively determine, repeatedly in real-time by the collaborationserver, a group intent vector from the plurality of received user intentvectors, and determine, repeatedly in real-time by the collaborationserver, an updated coordinate location based on the group intent vector;determine, repeatedly in real-time by the collaboration server, at leastone vehicle command from the group intent vector; send, repeatedly inreal-time by the collaboration server, the at least one a vehiclecommand to the vehicle controller, whereby the vehicle operates inresponse to the at least one vehicle command; send, repeatedly inreal-time by the collaboration server, the updated coordinate locationto the plurality of computing devices; and update, repeatedly inreal-time by the computing devices, the location of the collaborativelycontrolled graphical indication using the received updated coordinatelocation, whereby the coordinate location is updated each time the groupintent vector is determined by the collaboration server, whereby aclosed-loop system is formed between the collaboration server and eachcollaborative intent application.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features and advantages of severalembodiments of the present invention will be more apparent from thefollowing more particular description thereof, presented in conjunctionwith the following drawings.

FIG. 1 is a schematic diagram of an exemplary real-time vehicleoperation collaboration system.

FIG. 2 is a schematic diagram of a computing device in one embodiment ofthe vehicle operation collaboration system.

FIG. 3 is an exemplary user interface display of one computing device ofthe vehicle operation collaboration system.

FIG. 4 is an exemplary graphical magnet pointer of the interface displayin accordance with one embodiment of the present invention.

FIG. 5 comprises the exemplary graphical magnet pointer with a magneticon in a first position.

FIG. 6 comprises the exemplary graphical magnet pointer interface withthe magnet icon in a second position.

FIG. 7 comprises the exemplary graphical magnet pointer with the magneticon in a third position.

FIG. 8 is a flowchart of a method for real-time collaborative control ofa vehicle using a vehicle operation collaboration system

FIGS. 9-17 are exemplary user interface displays of one computing deviceof the vehicle operation collaboration system during collaborativecontrol of the vehicle.

FIG. 18 is a schematic diagram of an exemplary multi-swarm vehicleoperation collaboration system.

Corresponding reference characters indicate corresponding componentsthroughout the several views of the drawings. Skilled artisans willappreciate that elements in the figures are illustrated for simplicityand clarity and have not necessarily been drawn to scale. For example,the dimensions of some of the elements in the figures may be exaggeratedrelative to other elements to help to improve understanding of variousembodiments of the present invention. Also, common but well-understoodelements that are useful or necessary in a commercially feasibleembodiment are often not depicted in order to facilitate a lessobstructed view of these various embodiments of the present invention.

DETAILED DESCRIPTION

The following description is not to be taken in a limiting sense, but ismade merely for the purpose of describing the general principles ofexemplary embodiments. The scope of the invention should be determinedwith reference to the claims.

Reference throughout this specification to “one embodiment,” “anembodiment,” or similar language means that a particular feature,structure, or characteristic described in connection with the embodimentis included in at least one embodiment of the present invention. Thus,appearances of the phrases “in one embodiment,” “in an embodiment,” andsimilar language throughout this specification may, but do notnecessarily, all refer to the same embodiment.

Furthermore, the described features, structures, or characteristics ofthe invention may be combined in any suitable manner in one or moreembodiments. In the following description, numerous specific details areprovided, such as examples of programming, software modules, userselections, network transactions, database queries, database structures,hardware modules, hardware circuits, hardware chips, etc., to provide athorough understanding of embodiments of the invention. One skilled inthe relevant art will recognize, however, that the invention can bepracticed without one or more of the specific details, or with othermethods, components, materials, and so forth. In other instances,well-known structures, materials, or operations are not shown ordescribed in detail to avoid obscuring aspects of the invention.

As referred to in this specification, “media items” refers to video,audio, streaming and any combination thereof. In addition, the audiosubsystem is envisioned to optionally include features such as graphicequalization, volume, balance, fading, base and treble controls,surround sound emulation, and noise reduction. One skilled in therelevant art will appreciate that the above cited list of file formatsis not intended to be all inclusive.

Real-time occurrences as referenced herein are those that aresubstantially current within the context of human perception andreaction.

As described in related patent applications Ser. Nos. 14/668,970 and14/708,038, the massive connectivity provided by the Internet is used tocreate a real-time closed-loop emergent intelligence by collectingreal-time input from large numbers of people through a novel userinterface and processing the collected input from that large number ofusers into a singular group intelligence that can control a roboticdrone in synchrony, guiding the motion of the robotic drone andoptionally controlling the pan or tilt or zoom of a camera on a drone,in real-time. The methods use intervening software and hardware tomoderate the process, closing the loop around the disparate input fromeach of the many individual participants and the singular output of thegroup.

In one embodiment, each of a plurality of individual users(“participants”) engages a user interface on their own portablecomputing device 104, thereby conveying his or her individual real-timewill with respect to the desired motion of a drone in response toviewing the real-time video stream from the drone on a screen of saidportable computing device. The video stream is communicated to theportable computing device of each participant over a wirelesscommunication network. A swarm intelligence system combines thereal-time input from the plurality of users, closing a feedback looparound the plurality, enabling real-time control of the drone byemergent group intelligence.

Referring first to FIG. 1, a schematic diagram of an exemplary vehicleoperation collaboration system 100 is shown. Shown are a CentralCollaboration Server (CC) 102, a plurality of computing devices 104, adrone 105, a plurality of exchanges of data 106 with the CentralCollaboration Server 102, a drone controller unit 110, a collaborationmediation application (CMA) 112, a server communication link 114, acamera 115, a local controller 120, a drone communication link 122, anda collaborative intent application (CIA) 144.

The server communication link 114 (typically wireless, but in someembodiments may be wired) sends signals between the CCS 102 and theDrone Controller unit 110. The drone controller unit 110 (also referredto as the drone controller) receives commands from the CCS 102,processes those commands, and sends control signals over the wirelessdrone communication link 122 to the drone 105. The drone 105 includesthe local controller 120 that interprets the control signals from thedrone controller 110, and controls the actuators on the drone 105 toguide the drone 105 accordingly. In the case of a flying drone 105, theactuators are generally electronic motors controlling propellers. Alsoon the drone 105 is the camera 115 which may be fixed with respect tothe body of the drone 105, or may have actuators that enable pan andtilt motion of the camera 115 with respect to the body of the drone 105.The control signals from the drone controller 110 over the dronecommunication link 122 represent the desired position and/or motion ofthe drone 105 and represent the desired position and/or motion of thecamera 115 with respect to the drone 105 body. The control signals mayalso control other camera functions such as the zoom of a camera lens onthe camera 115. In this way, commands from the CCS 102 are enabled tocontrol the position, motion, speed, acceleration, altitude,orientation, and/or other dynamic characteristics of the drone 105, aswell as control the pan and tilt and zoom of the camera 115 affixed tothe drone 105. And in the case of drones 105 that include payloads orend effectors, the CCS 102 can be configured to send commands to thedrone controller 110 that indicate how the payload and/or end effectorsare to be manipulated by drone actuators.

In addition, the video captured by the camera 115 on the drone 105 iscommunicated over the plurality of exchanges of data 106 (typically viaa wireless network) such that it's streamed to the plurality of portablecomputing devices 104, in real-time and/or substantially near real-time.The video signal from the camera generally gets processed by drone LocalController 120, gets communicated over the drone communication link 122to Drone Controller 110, and either gets streamed directly to eachportable computing device 104, from the Drone Controller 110 (viaoptional communication link 124), or gets relayed to each portablecomputing device 104 through the CCS 102 via the exchanges of data 106.In this way, each user of each computing device 104 is enabled to (a)view the real-time first person video stream captured by the dronecamera 115, and (b) provide real-time input through a user interface 224on computing device 104, indicating a user intent with respect to thedesired motion of the drone and/or control of the drone camera or otherend effector.

Embodiments of the plurality of computing devices 104 and theinteraction of the computing devices 104 with the CCS 102 are previouslydisclosed in the related patent applications, indicating the system andmethods by which a plurality of distributed users, each of whom uses acomputing device 104, is able to provide real-time input as part of aclosed-loop control system, and thereby collaboratively express aunified intent as part of an emergent group intelligence. In the presentdisclosure, the unified intent, enables the plurality of users toexpress a singular desired motion (e.g. direction, speed, acceleration,altitude, orientation, or other dynamic characteristics) of the drone105. In the present disclosure the unified intent may optionally enablethe plurality of users to express a singular desired camera control(e.g. pan, tilt, and/or zoom) of the camera 115 on the drone 105.

More specifically, the collaboration system 100 comprises the CentralCollaboration Server (CCS) 102 that is running the collaborationmediation application 112 and is in communication with the plurality ofcomputing devices 104, each computing device 104 running theCollaborative Intent Application (CIA) 144, such that the plurality ofindividual users, each user interacting with one of the plurality ofcomputing devices 104, can provide user input representing a user intent(i.e. the will of the user) with respect to a desired motion of thedrone 105 (direction, position, speed, orientation, and/or altitude, etc. . . ) and/or a desired configuration of the camera (pan, tilt, and/orzoom). The plurality of user inputs is combined as a control system toresult in a group intent, thus enabling collaborative control of thereal-time graphical pointer 320, which enables the emergent will tocontrol the drone 105.

Referring next to FIG. 2, a schematic diagram of the computing device104 in one embodiment of the collaboration system is shown. Shown are acentral processor 202, a main memory 204, a timing circuit 206, adisplay interface 208, a display 210, a secondary memory subsystem 212,a hard disk drive 214, a removable storage drive 216, a logical mediastorage drive 218, a removable storage unit 220, a communicationsinterface 222, a user interface 224, a transceiver 226, an auxiliaryinterface 228, an auxiliary I/O port 230, communications infrastructure232, an audio subsystem 234, a microphone 236, headphones 238, a tiltsensor 240, the central collaboration server 102, and the collaborativeintent application 144.

As shown previously in FIG. 1, each of the plurality of computingdevices 104, each used by one of a plurality of users (the plurality ofusers also referred to as a group), is networked in real-time to thecentral collaboration server (CCS) 102. In some embodiments, one of thecomputing devices 104 could act as the central collaboration server 142.For the purposes of this disclosure, the central collaboration server102 is its own computer system in a remote location, and not thecomputing device 104 of one of the users. Hence the collaboration systemis comprised of the centralized central collaboration server 102 and theplurality of computing devices 104, each of the computing devices 104used by one user.

The computing device 104 may be embodied as a handheld unit, a pockethoused unit, a body worn unit, or other portable unit that is generallymaintained on the person of a user. In other embodiments the computingdevice may be a generally stationary device such as a desktop computeror workstation. The computing device 104 may be wearable, such astransmissive display glasses.

The central processor 202 is provided to interpret and execute logicalinstructions stored in the main memory 204. The main memory 204 is theprimary general purpose storage area for instructions and data to beprocessed by the central processor 202. The main memory 204 is used inthe broadest sense and may include RAM, EEPROM and ROM. The timingcircuit 206 is provided to coordinate activities within the computingdevice 204. The central processor 202, main memory 204 and timingcircuit 206 are directly coupled to the communications infrastructure232. The central processor 202 may be configured to run a variety ofapplications, including for example phone and address book applications,media storage and play applications, gaming applications, clock andtiming applications, phone and email and text messaging and chat andother communication applications. The central processor 202 is alsoconfigured to run at least one Collaborative intent Application (CIA)144. The Collaborative Intent Application 144 may be a standaloneapplication or may be a component of an application that also runs uponother networked processors.

The computing device 104 includes the communications infrastructure 232used to transfer data, memory addresses where data items are to be foundand control signals among the various components and subsystems of thecomputing device 104.

The display interface 208 is provided upon the computing device 104 todrive the display 210 associated with the computing device 104. Thedisplay interface 108 is electrically coupled to the communicationsinfrastructure 232 and provides signals to the display 210 for visuallyoutputting both graphics and alphanumeric characters. The displayinterface 208 may include a dedicated graphics processor and memory tosupport the displaying of graphics intensive media. The display 210 maybe of any type (e.g., cathode ray tube, gas plasma) but in mostcircumstances will usually be a solid state device such as liquidcrystal display. The display 210 may include a touch screen capability,allowing manual input as well as graphical display.

Affixed to the display 210, directly or indirectly, may be the tiltsensor 240 (accelerometer or other effective technology) that detectsthe physical orientation of the display 210. The tilt sensor 240 is alsocoupled to the central processor 202 so that input conveyed via the tiltsensor 240 is transferred to the central processor 202. The tilt sensor240 provides input to the Collaborative Intent Application 144. Otherinput methods may include eye tracking, voice input, and/or manipulanduminput.

The secondary memory subsystem 212 is provided which houses retrievablestorage units such as the hard disk drive 214 and the removable storagedrive 216. Optional storage units such as the logical media storagedrive 218 and the removable storage unit 218 may also be included. Theremovable storage drive 216 may be a replaceable hard drive, opticalmedia storage drive or a solid state flash RAM device. The logical mediastorage drive 218 may be a flash RAM device, EEPROM encoded withplayable media, or optical storage media (CD, DVD). The removablestorage unit 220 may be logical, optical or of an electromechanical(hard disk) design.

The communications interface 222 subsystem is provided which allows forstandardized electrical connection of peripheral devices to thecommunications infrastructure 232 including, serial, parallel, USB, andFirewire connectivity. For example, the user interface 224 and thetransceiver 226 are electrically coupled to the communicationsinfrastructure 232 via the communications interface 222. For purposes ofthis disclosure, the term user interface 224 includes the hardware andoperating software by which the user executes procedures on thecomputing device 104 and the means by which the computing device 104conveys information to the user. In the present invention, the userinterface 224 is controlled by the CIA 144 and is configured to displayinformation regarding the group collaboration, as well as receive userinput and display group output.

To accommodate non-standardized communications interfaces (i.e.,proprietary), the optional separate auxiliary interface 228 and theauxiliary I/O port 230 are provided to couple proprietary peripheraldevices to the communications infrastructure 232. The transceiver 226facilitates the remote exchange of data and synchronizing signalsbetween the computing device 104 and the Central Collaboration Server102. The transceiver 226 could also be used to enable communicationamong a plurality of computing devices 104 used by other participants.In some embodiments, one of the computing devices 104 acts as theCentral Collaboration Server 102, although the ideal embodiment uses adedicated server for this purpose. In one embodiment the transceiver 226is a radio frequency type normally associated with computer networks forexample, wireless computer networks based on BlueTooth® or the variousIEEE standards 802.11.sub.x., where x denotes the various present andevolving wireless computing standards. In some embodiments the computingdevices 104 establish an ad hock network between and among them, as witha BlueTooth® communication technology.

It should be noted that any prevailing wireless communication standardmay be employed to enable the plurality of computing devices 104 toexchange data and thereby engage in a collaborative consciousnessprocess. For example, digital cellular communications formats compatiblewith for example GSM, 3G, 4G, and evolving cellular communicationsstandards. Both peer-to-peer (PPP) and client-server models areenvisioned for implementation of the invention. In a third alternativeembodiment, the transceiver 226 may include hybrids of computercommunications standards, cellular standards and evolving satelliteradio standards.

The audio subsystem 234 is provided and electrically coupled to thecommunications infrastructure 232. The audio subsystem 134 is configuredfor the playback and recording of digital media, for example, multi ormultimedia encoded in any of the exemplary formats MP3, AVI, WAV, MPG,QT, WMA, AIFF, AU, RAM, Ra, MOV, MIDI, etc.

The audio subsystem 234 in one embodiment includes the microphone 236which is used for the detection and capture of vocal utterances fromthat unit's user. In this way the user may issue a suggestion as averbal utterance. The computing device 104 may then capture the verbalutterance, digitize the utterance, and convey the utterance to other ofsaid plurality of users by sending it to their respective computingdevices 104 over the intervening network. In this way, the user mayconvey a suggestion verbally and have the suggestion conveyed as verbalaudio content to other users. It should be noted that if the users arein close physical proximity the suggestion may be conveyed verballywithout the need for conveying it through an electronic media. The usermay simply speak the suggestion to the other members of the group whoare in close listening range. Those users may then accept or reject thesuggestion using their computing devices 100 and taking advantage of thetallying, processing, and electronic decision determination andcommunication processes disclosed herein. In this way the presentinvention may act as a supportive supplement that is seamlesslyintegrated into a direct face to face conversation held among a group ofusers.

For embodiments that do include the microphone 236, it may beincorporated within the casing of the computing device 104 or may beremotely located elsewhere upon a body of the user and is connected tothe computing device 104 by a wired or wireless link. Sound signals frommicrophone 236 are generally captured as analog audio signals andconverted to digital form by an analog to digital converter or othersimilar component and/or process. A digital signal is thereby providedto the processor 202 of the computing device 104, the digital signalrepresenting the audio content captured by microphone 236. In someembodiments the microphone 236 is local to the headphones 238 or otherhead-worn component of the user. In some embodiments the microphone 236is interfaced to the computing device 104 by a Bluetooth® link. In someembodiments the microphone 236 comprises a plurality of microphoneelements. This can allow users to talk to each other, while engaging ina collaborative experience, making it more fun and social. Allowingusers to talk to each other could also be distracting and could be notallowed.

The audio subsystem 234 generally also includes headphones 238 (or othersimilar personalized audio presentation units that display audio contentto the ears of a user). The headphones 238 may be connected by wired orwireless connections. In some embodiments the headphones 238 areinterfaced to the computing device 104 by the Bluetooth® communicationlink.

The computing device 104 includes an operating system, the necessaryhardware and software drivers necessary to fully utilize the devicescoupled to the communications infrastructure 232, media playback andrecording applications and at least one Collaborative Intent Application144 operatively loaded into main memory 204, which is designed todisplay information to a user, collect input from that user, andcommunicate in real-time with the Central Collaboration Server 102.Optionally, the computing device 104 is envisioned to include at leastone remote authentication application, one or more cryptographyapplications capable of performing symmetric and asymmetriccryptographic functions, and secure messaging software. Optionally, thecomputing device 104 may be disposed in a portable form factor to becarried by a user.

Referring next to FIG. 3, an exemplary user interface display 300 of onecomputing device 104 of the drone control system 100 is shown in oneembodiment of the present invention. Shown are a live video feed display305, a first control axis 310, a second control axis 335, acollaboratively controlled graphical pointer 320, a plurality of userobjects 330, a drone display area 340, a drone information area 350, achat area 360, a plurality of axis labels 370, a user input area 380,and a user intent vector indication 390.

The Collaborative Intent Application (CIA) 144, running on each PortableComputing Device 104, presents the user of that computing device 104with a unique interactive user interface display 300 which enables ashared experience with other users where each user can view the samelive video feed from the drone 105 and collaboratively control the drone105 by forming a closed-loop collective intelligence system. As shown inFIG. 3, the user interface display 300 includes the drone display area340 which includes in the present embodiment the live video feed 305,the drone information area, and the chat area 360. The live video feed305 presents substantially live video from the drone such that all usersin the system can see the same substantially real-time view from thedrone camera 115. As also shown, user interface display 300 optionallyincludes the a drone information area 350, which displays real-time dataabout the drone 105 and/or the collaborative group, such as includingthe speed, altitude, direction, and number of users currentlyparticipating in the real-time system 100. As also shown in FIG. 3, theuser interface display 300 may optionally include the live chat area 360wherein the participants can type and view chat comments, enabling textcommunication in near real-time among the participants.

As also shown in FIG. 3, the user interface display 300 includes theuser input area 380, a unique implantation of the closed-loop swarmintelligence interface disclosed in the related patents by the presentinventor which have been incorporated herein by reference. In thisinventive embodiment, the user input area 380 includes thecollaboratively controlled pointer 320. In the present embodiment, thepointer 320 is in the shape of a glass puck, although other graphicembodiments are contemplated. The location of the pointer 320 withrespect to the control axes 310, 335 is received from the CCS 102 asdetermined from the input of the collective group.

Also shown in the user input area 380 are a pair of intersecting controlaxes 310, 335, intersecting at the midpoint of the axes 310, 335. Asshown, the first control axis 310 depicts Forward-Backwards along avertical axis (as indicated by axis labels 370), and the second controlaxis 335 depicts Left-Right along a horizontal axis (as indicated byaxis labels 370). As also shown, each user in the system 100 associatedwith and enabled to manipulate (and thereby provide user input via) oneuser object 330 from the group of user objects 330, which in this caseis an the shape or a magnet. The user objects 330 controlled by otherusers are optionally shown in real time. In the moment of time depictedin FIG. 3, each user object 330 location relative to the pointer 320represents the user input of that user with respect to thecollaboratively controlled pointer 320, indicating that user's intentwith respect to the motion of the collaboratively controlled pointer320.

In this way, the Collaborative Intent Application 144 on each computingdevice 104 presents the user interface display 330 on the screen of eachComputing Device 104 such that each user can view and interact with thecollaboratively controlled pointer 320 (or other graphicalrepresentation of the group intent) that is manipulated by the group toposition the pointer 320 relative to the control axes 310, 335 toindicate a direction of motion of the drone 105, and a magnitude forthat direction of motion that is translated into speed of the drone 105.In this way, if the real-time closed-loop system of users moves thecollaboratively controlled graphical pointer 320 to a position on thecontrol axes 310, 335, that position indicates if the drone 105 shouldmove forward, back, left, right, or some combination thereof, and atwhat relative speed it should move in the forward, backward, leftward,rightward, or combinatorial direction.

The computing devices 104 are in communication with the CCS 102 as shownby the data exchanges 106 an FIG. 1. In some embodiments, such as amulti-tier architecture, the computing devices 104 may communicate witheach other. The CCS 102 includes software and additional elements asnecessary to perform the required functions. In than application, itwill be understood that the term “CCS” may be used to refer to thecollaborative mediation application 112 of the CCS 102 or other elementsof the CCS 102 that are performing the given function.

By collaboratively moving the pointer 320, said plurality of users isenabled to impart real-time directional intent for the motion of thedrone. The user input is expressed in real-time, at every time step, byeach user manipulating the corresponding user object 330 (i.e. thegraphical magnet) by using a touch screen or mouse or other appropriatecontroller. The user positions the user object 330 (i.e. magnet) at alocation and orientation with respect to the collaboratively controlledpointer 320 (i.e. puck) and thereby indicates a user intent vector,including both a direction and magnitude of the user input. Oneexemplary user intent vector indication 390 is shown in FIG. 3 (forclarity, only one user intent vector indication is shown, but it will beunderstood that each user object 330 is associated with one user intentvector). The user intent vector can optionally be input by the user, forexample, by tilting his or her computing device 104 in the desireddirection. In other embodiments the user intent vector is optionallyinput by repeatedly swiping on a touchscreen at a direction, and at aswipe rate, that indicates the user intent vector. The user intentvector is communicated by the CIA 144 running on the user's computingdevice 104, to the Central Collaboration Server (CCS) 102. The CCS 102receives the user intent vectors from the plurality of users, and thenderives a group intent vector from the plurality of user intent vectorsthat represents the collective will of the group at that time step.

The group intent vector is then used to compute an updated location ofthe pointer 320 with respect to the control axes 310 335. This updatedlocation is translated into a signal for controlling the drone 105. Thefurther the pointer 320 is along one axis, the faster the drone 105 iscommanded to move in the direction that corresponds to that axis. So,the further away from the origin of the control axes 310 335 that thepuck 320 moves along the direction of the first control axis 310 towardsthe axis label 370 labeled “FORWARD”, the faster the drone 105 moves inthe forward direction. Similarly, the further from from the origin ofthe control axes 310 335 that the puck 320 moves along the secondcontrol axis 335 towards the axis label 370 labeled “RIGHT”, the fasterthe drone 105 is commanded to move in the rightward direction. It shouldbe noted that the axes 310 335 shown are relative to the first-personview presented by the camera 115. In other embodiments, the control axes310 335 can be configured in ordinal directions such as NORTH, SOUTH,EAST, or WEST. It should also be noted that because the axes 310 335 aretwo-dimensional, the real-time closed-loop system of users can positionthe collaboratively controlled pointer 320 to locations on the ControlAxes 310 335 that simultaneously indicate a speed in two differentdirections at once, thus giving a vector direction. For example, if thepuck 320 is moved to a position that is away from the origin towardsboth the FORWARD and RIGHT axes labels 370 at once, the relativedistance of the collaboratively controlled pointer 320 from the originin the FORWARD direction will indicate a speed of the puck 320 in theforward direction, while the relative distance of the collaborativelycontrolled pointer 320 from the origin in the RIGHT direction willindicate a speed of the drone 105 in the rightward direction. In thisway, the group intent can indicate that the drone 105 should moveFORWARD and to the RIGHT, with a speed indicated by how far along eachaxis 310 335 the group has positioned the puck 320 by working togetheras a real-time collective intelligence.

As previously shown in FIG. 1, the system 100 comprises the CentralCollaboration Server 102 (“CCS”) that's in communication with theplurality computing devices 104, each of said computing devices 104running the Collaborative Intent Application (“CIA”) 144. The system 100is designed to enable a plurality of users, each engaging an interfaceof one of said computing devices 104, to jointly control the singlegraphical pointer 320 through real-time group-wise collaboration. Thegraphical pointer 320 (also referred to as a puck) is displayed to eachuser by the CIA 144 running on his or computing device 104, ascoordinated by data received from the CCS 102 over the communicationlinks 106. Each of the computing devices 104 comprises one or moreprocessors capable of running the CIA routines and displaying arepresentation of the pointer 320 along with a plurality of othergraphics. The computing device 104 could be, for example, a personalcomputer running the CIA 144 or a smart phone or tablet running the CIA144. The CIA 144 can be a stand-alone executable or be code thatexecutes inside a web-browser or other shell.

The CIA software 144 running on each computing device 104 is configuredto display a graphical user interface (also referred to as a userinterface display or a decoupled control interface) that includes atleast one graphical pointer 320 and a plurality of graphical indicationsfor user choices (referred to some embodiments as input choices oranswer choices). In some embodiments, the graphical pointer 320 isconfigured to look like a “puck” with a central viewing area that ispartially transparent. In the present embodiment, in lieu in individualinput choices, the control axes 310, 335 are used to indicate user inputchoices. When the pointer 320 is positioned over a location with respectto the control axes 310, 335 for more than a threshold amount of time, avalue associated with that location is selected for each of the controlaxes 310, 335. These values are then translated into a command sent bythe CCS 102 to the Drone Controller 110, indicating the desired motionof the drone 105, based on the location of the collaborativelycontrolled puck 320 on the control axes 310, 335. In response The DroneController 110 then sends a corresponding command to the drone 105,which is processed by the local controller 220 on the drone 105, whichthen modulates the drone actuators (motors controlling propellers) to beresponsive to commands from the CCS 102. In this way, the real-timecollective intelligence of users of the plurality of computing devices104, can work together as a system to position the puck 320 on thecontrol axes 310, 335, and thereby send commands to the drone 105 thatcorrespond with the desired speed and direction indicated on the controlaxes 310, 335. In other embodiments, a third axis can be used to alsoindicate orientation of the drone 105 based on the position ororientation of the controlled graphical puck 320 on the control axes.

As described in detail in the aforementioned related patentapplications, the CIA 144/CCS 102 software enables the group of users toeach impart their own individual input so as to collaboratively controlthe motion of the graphical pointer 320, said pointer 320 moving undergroup-wise control. In a preferred embodiment of the present invention,a physically intuitive metaphor is employed such that the pointer 320 isassigned a simulated mass and a simulated damping (friction) withrespect to the control axes 310. Each user is told that their personaluser input acts to apply a simulated force upon the group-wise pointer320. This is represented as the user intent vector described in therelated applications. The user intent vector can be applied bymanipulating a user object 330, such as the graphical magnet 330, eachof which is freely controllable in real-time by the corresponding user,enabling the user objects 330 to be positioned and oriented with respectto the pointer 320. The pointer 320 then moves in response to anintelligent summation of the applied forces. In some embodiments,artificial gravity is also applied to the puck 320, pulling the puck 320back to the center of the control axes. The gravitational force is thuspart of the physical system that also includes the user intent vectorforces. The vector sum can be a simple vector addition in which eachuser input is counted equally, or it can be a weighted sum (or average)in which the input from some users has more impact than others. Asdescribed in the related applications, the weighting process can bebased on user scores earned during previous sessions or based onreal-time behavioral characteristics. In some embodiments, as disclosedin related patent application Ser. No. 15/898,468 by the presentinventor, the weighting factors can be computed based on predictivemeasures of user confidence and can be optimized using Machine Learningtechniques.

Thus the intuitive conceptual model is provided to users wherein theplurality of user force vectors are applied to the pointer 320 basedupon input conveyed by each user into their individual computing device104. This is achieved by computing and imparting the group force vectorupon the pointer 320 that is an intelligent summation or average of theuser input force vectors. The computing and imparting is performed theCCS 102 which collects the real-time input from the users, computes theresultant vector, and applies it to a physics-based model controllingthe graphical movement of the displayed pointer 320 with respect to thecontrol axes 310, 335. The physics-based model considers a pointer mass,a environmental damping coefficient, and a current vector motion(velocity and acceleration) of the pointer 320, and determines anupdated vector motion of the pointer 320 resulting from the currentgroup force vector.

Because the users are continually providing user inputs, the group forcevector is repeatedly calculated at time steps faster than humanperception can detect. At similar speeds, the group force vector isrepeatedly applied and the vector motion of the pointer 320 repeatedlyupdated. This creates a seamless closed-loop control system connectingall of the participating users. In some embodiments, this is performedat rates in which updates are performed every 100 milliseconds, with alag of less than 25 milliseconds. In many such embodiments pointermotion is interpolated between updates based on the physics model. Evenwhen no forces are applied by the users, the pointer 320 generallymaintains momentum and will continue to move for a period of time beforestopped by damping. This enables a smooth and intuitive system whereinparticipants connected from around the world can work together insynchrony to control the pointer 320, and thereby control the drone 105,together as a unified intelligence.

Providing the intuitive conceptual model for group-wise control of thesingle pointer 320 is helpful, but smooth and seamless operation hasrequired an intuitive graphical user interface and control methodology.The challenge of the pointer interface is that unlike traditional userinterfaces where a user's action has a direct and apparent impact on theobject they are intending to control (e.g. the pointer 320), thiscollaborative system 100 is such that the motion of the pointer 320 isnot based on the user's own input but is based on the group input.Because of this, the user may impart a desire for the pointer 320 tomove left at a given moment, but if the group intent is determined fromthe group of users as a desire for the pointer 320 to move right, thepointer 320 will move right. Thus, a significant need exists forintuitive graphical user interface methodologies that allow theindividual user to see a result of his or her input, while also makingthe overall physical metaphor as clear and simple and intuitive aspossible. More specifically, there is a substantial need to create a newtype of user interface that intuitively connects but also substantiallydecouples the representation of each user's personal input from themotion of the collaborative controlled pointer 320. Some embodiments ofintuitive graphical user interface methodologies have been described inthe related patent applications.

Referring next to FIG. 4, as previously disclosed in related patentapplication Ser. No. 14/738,768, a graphical representation of agraphical magnet pointer interface 900 is shown in one embodiment of anintuitive graphical interface methodology. Shown are a pointer 400, apointer vertical axis 402, a magnet icon 404, a magnet axis 406, anangle 408, and a pointer center 410. The pointer 400 may be an exampleof, or incorporate aspects of, the pointer 320 as described withreference to FIGS. 3 and 9-15. The magnet icon 404 may be an example of,or incorporate aspects of, magnet icons 330 as described with referenceto FIGS. 3, 10-13, 15 and 16.

The graphical magnet pointer interface illustrated in FIGS. 4-7 is amethodology for user input that supports a physically intuitive modelfor group-wise control of the graphical pointer 400. It employs themagnet icon 404 that is provided to each user for display on theircomputing device 104 (as controlled by the instance of the CIA 144software running on the user's computing device 104). In the embodimentshown, the magnet icon 404 is a “U” shaped magnet icon, but other typesof magnet icons can be used, and/or other elements that graphicallyrepresent a physical pull force. In this way, each user can see his ownmagnet icon 404 on his own screen, said magnet icon 904 being directlyresponsive to the user input provided by said user. Because the controlof the magnet icon 404 is handled locally by the computing device 104,the graphical magnet pointer interface is highly responsive and notimpacted by communication lag with the CCS 102, thus allowing each userto feel like he has a high-bandwidth highly responsive link into thesystem 100. The position of the magnet icon 404 on the user's interfacedisplay (for example the exemplary user interface display 300) may becontrolled by a mouse coupled to the computing device 104 and operatedby the user, with a conventional mouse arrow icon changing to the magneticon 404 when the mouse cursor nears the graphical pointer 400 that isalso displayed on the user interface display. The magnet icon 404 isdisplayed at the location of the mouse arrow icon, but is configured inthe software to always point towards the center 410 of the circularpointer 400. Thus as the magnet icon 404 approaches the pointer 400, themagnet icon 404 appears to aim at the pointer center 410 as if themagnet icon 404 is magnetically attracted to the pointer 400.

In addition, the CIA 144 software controlling the magnet icon 404 may beconfigured to increase a size of the magnet icon 404 as the magnet icon404 moves closer to the pointer 400, corresponding to a larger“magnetic” force between the magnet icon 404 and the pointer 400. Thus,with a very intuitive graphical metaphor, the user understands withoutinstruction that he can apply a virtual pull force on the pointer 400(representing his user intent vector) that aims from the pointer center410 to the location of the cursor (i.e. the magnet icon 404) controlledby the mouse.

As shown in FIG. 4, the CIA 144 software can be configured on eachcomputing device 104 to display the graphical magnet icon 404 at thecursor location controlled by the user's mouse, trackpad, trackball,touchscreen, or other means for user input of the computing device 104.The magnet icon 404 is configured to appear only when the user's cursorlocation is within threshold proximity of the group-controlled pointer400, for that means the user intends to convey user input regarding hispersonal intent as to which direction the pointer 400 should move. Themagnet icon 404 is configured to automatically point towards the pointercenter 410, as if magnetically attracted to it. This conveys anintuitive feeling to the user that by positioning the magnet icon 404near the pointer 400, he is applying the magnetic pull on the pointer400. The CIA 144 software then sends the user intent vector to the CCS102 indicating the angle 408 of the magnetic pull. In the embodimentshown in FIG. 4, the longitudinal magnet axis 406 of the magnet icon 404is the axis of symmetry of the singly-symmetric magnet icon 404. Theangle of direction is the clockwise angle 408 between the pointervertical axis 402 and the magnet axis 406. In the example shown in FIG.4, the angle 408 is approximately 330 degrees.

In some embodiments, magnitude of the user input can be graphicallyconveyed by how close or far the user positions the magnet icon 404 tothe pointer 400. The closer the magnet icon 404 to the pointer center410, the stronger the magnitude of the user input (i.e. the “magneticforce”). To make this visually intuitive, the magnet icon 404 increasesin size as the magnet icon 404 moves closer to the pointer center 410.Once the magnet icon 404 overlaps the pointer 400, the magnet icon 404may be limited from getting too close the pointer center 410 (i.e. fromcovering a central targeting area of the pointer 400). Thus the magneticon 404 appears when the input cursor gets within certain proximity ofthe pointer 400, increases in size as the cursor nears the pointer 400,and disappears if the cursor gets too close to the pointer center 410,the magnet icon size increasing as the magnet icon 404 moves closer tothe pointer center 410.

Referring next to FIGS. 5-7, exemplary magnet icon pointerconfigurations are shown. Shown in FIG. 5 is a first magnet pointerconfiguration. Shown in FIG. 6 a second magnet pointer configuration,and shown in FIG. 7 a third magnet pointer configuration. FIGS. 5-7 showexample configurations of the magnet icon 404 and associated pointer 400at various points in time. The pointer 400 will be moving across thedisplay 210 (for example, the user interface display 300 of FIG. 3)based on the group intent as determined by the CCS 102 and sent to eachcomputing device 104 to be displayed on the display 210 by the CIA 144.The magnet icon 404 is controlled by the user input in conjunction withthe CIA software residing on the computing device 104, the magnet icon404 representing the desired magnitude (size of magnet) and direction(position/orientation of magnet) of the user intent. Based on themagnitude/direction/position of the magnet icon 404 with respect to thepointer 400, the CIA 144 software sends the user intent vector to theCCS 102 for use in deriving the next group intent.

As shown in FIG. 5, a first magnet icon 502 has a medium size comparedto a second magnet icon 602 of FIG. 6 and a third magnet icon 702 ofFIG. 7. This indicates a medium magnitude of intent. The first magneticon 502 has been located by the user in a direction as defined by thefirst angle 500, shown to be approximately 50 degrees.

As shown in FIG. 6, the size of the second magnet icon 602 is smallerthan the first magnet icon 502 and the third magnet icon 702, indicatinga relatively small magnitude of user intent. The second magnet icon 602has been located by the user in a direction defined by the second angle600, shown to be approximately 230 degrees.

As shown in FIG. 7, the size of the third magnet icon 702 is larger thanthe first magnet icon 502 and the second magnet icon 602, indicating arelatively larger magnitude of user intent. The third magnet icon 702has been located by the user in a direction defined by the third angle700, shown to be approximately 310 degrees. The CCS 102 sums the userintent vectors from the plurality of users, computes the group intentvector, uses the group intent vector to apply the group force vector tothe simulated physical model of the pointer 400 (mass, damping, etc . .. ), and based on the physics model sends the pointer coordinateinformation to each computing device 104, each of which are then updatedwith the new location of the pointer 400.

The result is the satisfying, intuitive, informative, and fun method bywhich individual users can convey their intent/will upon the graphicalpointer 400 that is being controlled not by them individually, but bythe group of users who are all applying real-time synchronous controlinput.

As described previously, some embodiments weight the input from allusers equally. In such embodiments, the magnet icons 404 displayed forall individual users can employ the same mapping between size anddistance to the pointer 400. However, for embodiments that weight usersdifferently, magnet size can be scaled accordingly. In this way, theuser who is being granted a higher contribution rate to the group due toearning points, can see a larger magnet icon 404 on their screen thanthe user who has been granted a lower contribution rate to the group.This provides visual intuition.

In general, users only see their individual magnet icon 404 on theirscreen. In some embodiments, however, the system 100 can be configuredto allow the user to see a representation of the magnets controlled byother users. In such embodiments “ghost magnet” icons representing userinputs from other users are employed. The ghost magnet icons may bepartially transparent, thus making the ghost magnet icons easilydistinguishable from the user's own magnet icon, and thus preventing theghost magnet icons from obscuring other important elements on thedisplay 210. If the user is collaborating along with 100 other users,the user might thus see one solid magnet icon 404 (under his owncontrol) and 99 ghost magnet icons that represent the real-time userinput being conveyed by the other users. The ghost magnet icon for oneof the other users would only appear when that user is positioning hismouse near the representation of the pointer 400 on his display 210. Theghost magnet icons in some embodiments may resemble a swarm of bugshovering around the pointer 400. When all ghost magnet icons are evenlydistributed around the pointer 400 (accounting for both magnitude anddirection), the net effect cancels out and the pointer 400 does notmove. But as the group converges upon a unified sentiment, a majority ofthe magnet icons would be seen to group themselves on one side of thepointer 400, and the pointer 400 will move. Such a display helps toconvey the group-wise behavior of the users which in many ways emulatesswarms of bugs or flocks of birds. The ghost magnet paradigm is agraphical representation of this swarm-like behavior.

While the graphical magnet interfaces as shown in FIGS. 4-7 uses cursorcontrol interfaces such as mice, touchpads, trackballs, andtouchscreens, a number of unique tilt control input methods can beemployed as disclosed in the related applications incorporated herein byreference.

As disclosed previously, an artificial restoring force is applied to thepuck/pointer 400 in many inventive embodiments, pulling the puck 400back to the center of the control axes in resistance to the pull of themagnets of the users. This restoring force may be modeled as a linearspring force, a non-linear spring force, a gravitational well thatincreases the restoring force level with distance from the origin. Insome embodiments a graphical representation of the restoring force isshown on the user interface display.

Referring next to FIG. 8, a flowchart of a method or real-timecollaborative control of a vehicle using a vehicle operationcollaboration system (such as the collaboration system 100 shown inFIG. 1) is shown.

In the first display information and pointer step 800, the CCS 102 sendsinformation to the plurality of computing devices 104 such that an inputdisplay (for example the interface display 300 of FIG. 3) is shown oneach computing device. The display includes a collaboratively controlledgraphical pointer indication (such as the pointer 320 shown in FIG. 3).Other vehicle control information is shown such that locating thepointer in relation to the vehicle control information corresponds toone or more commands that can be send to the vehicle (or devicesattached to and controlled via the vehicle, such as a camera). In theexemplary interface display 300 of FIG. 3, the vehicle controlinformation includes the first control axis 310 representing vehiclecommands for forward/backwards motion, and the second control axis 335representing device commands for left/right vehicle motion.

In the next initial state step 802, the pointer location on the displayand the vehicle are at an initial state. In the embodiment shown in FIG.3, the initial state of the pointer is at the intersection (origin) ofthe normal control axes).

In the next start time step step 804, one vehicle command collaborativetime step is begun. As previously described, the pointer update timestep length may be different from the drone update time step length. Inother embodiments the time steps may be of the same length or the dronetime step may be shorter than the pointer update time step.

The method proceeds to the receive user inputs step 806, the beginningof the pointer update loop. During the receive user inputs step 806,each computing device 104 receives an input associated with a desiredlocation of the pointer (which in turns corresponds to a desired motionof the vehicle). Typically, the user input will comprise a user inputvector including a magnitude and a direction.

In the next determine group input step 808, each computing device 104sends the user input to the central collaboration server 102. Thecentral collaboration server determines from all user inputs a singlegroup input representing the combination of all user inputs. In someembodiments the group input comprises a group input vector having adirection and a magnitude.

In the next determine coordinate location step 810, the CCS 102determines a coordinate location based on the group input. Thecoordinate location is sent to each computing device 104 in the sendcoordinate location to computing devices step 812.

In the move pointer location step 814, in response to receiving thecoordinate location, each computing device updates the pointer locationbased on the coordinate location, therefore representing on eachcomputing device 104 the combined group result determined from theplurality of user inputs. The process then returns to the receive userinputs step 806, and continues to loop through the pointer update steps806, 808, 810, 812, 814. At certain times the process will continued tothe determine vehicle control commands step 816 for the process ofsending commands to the vehicle.

In the next determine vehicle control commands step 816, the CCS 102determines at least one vehicle command from the coordinate location. Inthe embodiments shown in FIGS. 9-12, the direction of motion of thedrone 105 is determined by the coordinate location as represented by thepointer 320 and the speed of the drone 105 is determined by the distanceof the pointer 320 from each of the control axes 310, 335 (for example,the forward distance 1200 and the rightward distance 1202 shown below inFIG. 12). In the next send vehicle commands step 820, the CCS 102 sendsthe command(s) to the drone controller 110, which in turn sends thecommand(s) to the drone 105. Then, in the next vehicle carries outcommands 820, the vehicle performs according to the received commands.

In the last end time step step 822, the method. returns to the starttime step step 804, and the process repeats in continuous loops, withthe users providing inputs and the CSS 102 determining drone commandsand updated pointer locations from combining the user inputs into asingle group input. Examples using this general method are shown belowin FIGS. 9-17.

Referring next to FIGS. 9-12, exemplary user interface displays 900 ofone computing device 104 of the drone control system 100 are shown atsequential points in time while a group of users is collaborativelycontrolling the pointer 320 as a unified system in one embodiment of thepresent invention. Shown are the first control axis 310, the secondcontrol axis 335, the collaboratively controlled graphical pointer 320,the plurality of user objects 330, the plurality of axis labels 370, andan origin 902.

In FIG. 9, an initial point in time is shown while the drone 105 is atan initial position and the puck 320 is at the center of the controlaxes 310, 335. This is a starting position wherein the control valuesare 0,0 (i.e. there is no movement along either axis 310, 335), and thusthe drone is currently being told to remain at a fixed location.

As described above, however, the graphical puck 320 is undercollaborative control from the plurality of networked users, the usersable to work as a system to move the puck 320 upon the control axes 310,335 and therefore guide the drone in two degrees of freedom—forward/backand left/right—where the speed in each degree of freedom is a functionof the displacement of the puck 320 with respect to the control axes310, 335. As also mentioned above, additional degrees of freedom may beemployed, for example an elevation axis (up/down).

As described above, users collaboratively control the puck 320 usinggraphical magnets icons that act, as part of a dynamic system. Referringto FIGS. 10-12, three sequential times step are shown in which the puck320 is being roved with respect to the control axes 310, 335 by theplurality of networked users controlling individual user objects 330(shown as magnet icons in these examples) on their computing devices104.

As shown in FIG. 10, each user positions his user object 330 around thepuck 320 and impart the individual user force vectors. The puck 320 mayalso have a gravitational (or spring) restoring force bringing it backto center. Thus the users must overcome the restoring force to move thepuck 320 away from the origin. This is a highly valuable inventiveembodiment because the center position is one in which the drone staysstill. Thus if users are not applying sufficient net force on thegraphical puck 320, the drone hovers in place, as shown in FIG. 10.

If the users apply a sufficient force to overcome the restoring force,the puck 320 will move away from the origin, as shown in FIG. 11,resulting in movement commands being sent to the drone 105 as previouslydescribed.

As the users continue to supply input, the puck 320 may move evenfurther from the origin as shown in FIG. 11. In FIG. 11, the puck 320 ispositioned such that the puck position corresponds with both FORWARDmotion along the first control axis 310 and and RIGHTWARD motion alongthe second control axis 335. The Central Collaboration Server 102 willsend commands to the Drone Controller 110 based on the changing positionof the graphical puck 320 under real-time collaborative control. At themoment shown in FIG. 12, the commands sent by the CCS 102 to the DroneController 110 will indicate that the drone should move FORWARD andRIGHTWARD, at a forward speed proportional to a forward distance 1200equal to the distance between a center of the puck 320 and the originalong the first control axis 335, and a rightward speed proportional toa rightward distance 1202 between the center of the puck 320 and theorigin along the first control axis 310. In this way, a group ofdistributed and networked users can work together to collaborativelycontrol both the direction and speed of the drone 105 in multipledegrees of freedom.

Referring next to FIG. 13 an exemplary user interface display 1300 ofone computing device 104 of the drone control system 100 during a timewhile a group of users is collaboratively controlling the pointer 320 asa unified system is shown in another embodiment of the presentinvention. Shown are the first control axis 310, the second control axis335, the graphical pointer 320, the plurality of user objects 330, theplurality of axis labels 370, and the origin 902.

As shown, the control axes 310, 335 can be employed for controlling thecamera pan, tilt, and zoom. An example is shown in FIG. 13. The firstcontrol axis 310 results in commands for forward/rearward tilting of thedrone camera 155, as displayed for the user by the axis labels 370 “TILTFORWARD” and “TILT BACK”. The second control axis 335 results incommands for panning left and right of the drone camera 155, asdisplayed for the user by the axis labels 370 “PAN LEFT” and “PANRIGHT”.

While FIG. 13 only corresponds to PAN and TILT control, alternative axescan be employed by the present invention, for example collaborative ZOOMcontrol. In addition, collaborative selection of options such as“RECORD” or “SNAPSHOT” may also be employed by the present invention.

The examples given thus far have enabled a swarm or users to control adrone through an interface that gives relative motion commands—directionand speed—in multiple degrees of freedom. In other embodiments of thepresent invention, direct POSITION CONTROL can be enabled in a 2dimensional area or 3 dimensional space. We will use the 2 dimensionalarea for illustrative purposes, but the same inventive methods andsystems can be used for additional dimensions of control.

Referring next to FIG. 14, an exemplary user interface display 1400 ofone computing device 104 of the drone control system 100 at a startingpoint in time while a group of users is collaboratively controlling thepointer 320 as a unified system is shown in another embodiment of thepresent invention. Shown are the first control axis 310, the secondcontrol axis 335, the graphical pointer 320, the plurality of axislabels 370, and the origin 902, a grid 1402, and a control area 1404

FIG. 14 shows an alternate interface display in which thecollaboratively controlled graphical pointer 320 (i.e. puck) isdisplayed on the two-dimensional control area 1404 (depicted as a set ofX and Y coordinates) that represents the position of the drone 105 in areal two-dimensional plan area in the physical world. The scaling of thecontrol area 1404 can be configured as adjustable or fixed. Thisrepresentation is shown on the portable computing device 104 of eachuser in the collaborative real-time control system, as generated by theCollaborative Intent Application (CIA) 144 running on the portablecomputing device 104 and in communication with the collaborative controlsoftware 112 running on the Central Collaboration Server 102. In oneembodiment, the two-dimensional control area 1404 overlaid by the grid1402 in FIG. 14 represents a 200 ft. by 200 ft. plan space over a largefield (for example a football field) that the drone is hovering over.The origin 902 of the grid 1402 in FIG. 14 corresponds with the centerof that 200 ft by 200 ft physical space and is the starting location ofthe drone 105 when the session starts.

As shown in FIG. 14, at the starting time the puck 320 is positioned atthe origin 902 of the grid 1402. Then, using the collaborative controlmethods disclosed herein, each user off the portable computing device104, while watching a live video feed from the drone 105, expressestheir real-time individual intent by repeatedly updating their desiredPULL on the puck 320 using the magnet method disclosed previously inFIGS. 4-7 and in related patents that have been incorporated byreference (for example, the screen-swipe method intended for mobilephones). The CCS software 112 closes the loop around the plurality ofusers, repeatedly updating the location of the collaborativelycontrollable pointer (puck) 320 with respect to the two dimensionalcontrol area 1404 shown in FIG. 14. The location of the puck 320 withinthe area 1404 is then translated into the control signal 114 to be sentby the CCS 102 to the Drone Controller 110, the control signalindicating a repeatedly updated target position for the drone 105 withinthe real-world physical space, the target position a scaledrepresentation of the position of the graphical puck 320 within thecontrol area shown in FIG. 14. In this way, as the group of users movesthe puck 320 as a real-time collective intelligence system, the drone105 will move within the real-world physical space.

Referring next to FIG. 15, the exemplary user interface display 1400 ofone computing device 104 of the drone control system 100 at anotherpoint in time while the group of users is collaboratively controllingthe pointer 320 as a unified system is shown in another embodiment ofthe present invention. Shown are the first control axis 310, the secondcontrol axis 335, the graphical pointer 320, the plurality of userobjects 330, the plurality of axis labels 370, the origin 902, the grid1402, and the control area 1404.

FIG. 15 shows an example time step in which the swarm of networked usersmoved the graphical puck 320 to a non-origin location within the controlarea 1404 of the user interface display 1400. As the puck 320 moved fromthe starting position (at the origin 902) to the position shown in FIG.15, the CCS 102 sent a stream of repeated updates to the DroneController 110, indicating a stream of new scaled locations in thephysical space for the drone 105 to move to. In many such embodiments,the collaboratively controlled graphical pointer 320 is updated at afast rate, while the drone's commanded position is updated at a lowerrate. The fast rate may be, for example, every 200 milliseconds, whilethe slow rate may be every 2 seconds.

A valuable additional inventive method of the present invention is theunique control methods by which the system maintains a closecorrespondence between the location of the puck 320 in the control area1404 (with respect to the reference origin 902) and the location of thedrone 105 in the real physical world, with respect to a real-worldreference location. Maintaining this correspondence is important becauseit takes time for the drone 105 to move to a new location in the realworld, and if the real-world space is large, that time can besubstantial. Conversely, moving a graphical pointer is usually very fastand could easily get way ahead of where the real drone actually is. Tocorrect for this mismatch, the following methods have been developed.

Method 1: Simulated Mass and Simulated Damping

The repeatedly updated location of the collaboratively controlledpointer 320 is based on a real-time physical control system run by theCCS server 102, the control system based on a physics model that assignssimulated mass and/or damping to the puck 320, governing its motion inresponse to the user applied forces. By assigning appropriate massand/or damping values, the simulated physical dynamics of the puck 320can be made to match the physical limits of the drone 105 in the realworld. For example, by assigning a sufficiently high mass and/or dampingto the puck 320, based on the responsiveness of the real drone in thereal world, the motion of the puck 320 can be sufficiently constrainedsuch that its location within the control area does not getsubstantially ahead of the drone's location within the real physicalspace. Thus, by performing “dynamic matching” of the physics model ofthe collaboratively controlled pointer 320 to the real physicalresponsiveness of the drone, the experience of the participants will besubstantially enhanced.

Method 2: Simulated Restoring Force

In addition to the mass and damping characteristics described in Method1 above, or instead of them, a simulated restoring force may be appliedto the puck 320, pulling the puck 320 back towards the location in thecontrol area 1404 that corresponds with the current location of thedrone in the real physical space. An example of the simulated restoringforce method is shown in FIG. 16. The location on the control area 1404corresponding to the actual current location of the drone 105 is thescaled virtual location of the drone (SVLD) 1600, as the location is ascaled representation of the location of the drone 105 in the realworld, mapped virtually to the control area 1404 upon which the puck 320is being manipulated. The restoring force 1602 pulls the puck 320 backtowards the SVLD location 1600 within the control area 1404. The usersmust overcome the restoring force 1602 to break the correspondencebetween the two spaces, and if the restoring force 1602 increases withincreasing distance between the current scaled virtual location 1600 ofthe drone and the current location of the puck 320, the size of themismatch will be maintained as very small. To enable this method, thedrone 105 must track its position, either using GPS or other coordinatetracking methods, or by dead reckoning based on motion. The drone 105then repeatedly sends an updated representation of its location to theCCS 102, which then compares the puck 320 location to the SLVD locationand applies the restoring force 1602 to the puck 320 accordingly.

The restoring force 1602 may be a linear or non-linear spring force, thegreater the distance between the puck 320 and the SVLD location 1600,the larger the restoring force 1602. By applying a simulated stiffspring, the puck 320 will not deviate far from the position in thecontrol area 1404 that corresponds with the actual location of the drone105. This offers an advantage to Method 1 above, in that the restoringforce method tracks the actual location of the drone 105, so even if thedrone 105 faces unexpected dynamics (for example a head wind or tailwind), the puck 320 and drone 105 do not deviate substantially.

In some embodiments of the present invention, the system may switchbetween relative motion control (as shown for example in FIG. 9) whereinthe users collaboratively control a direction and speed to the dronewithin a space, and absolute position control (as shown for example inFIG. 14) wherein the users collaboratively control the absolutelyposition of the drone in a space. Enabling these two alternate modes,and enabling a seamless switch between the modes, has substantialinventive value as the relative motion control mode is useful forcontrolling a drone as it moves over a large distance, and the absoluteposition control is useful for controlling a drone with fine positioningwithin an area. The current invention enables the users to selectivelyswitch between modes. Furthermore, when the users switch from relativecontrol mode, to absolute position mode, the origin of the control area(i.e. the center of the axes 310, 335 shown in FIG. 14)) is updated tothe current plan coordinate position of the drone 105 in the physicalworld.

Thus during a typical flight of the drone 105, the users may start outin relative control mode and may command the drone 105 to fly northwestat a particular speed, for a particular time, with continuous real-timeupdating of the direction and speed, enabling the drone to fly undercollaborative control of the users to a general location in the realworld. The users may then switch to absolute control mode. When thatswitch is made, the origin of the control area (i.e. the origin 902 asshown in FIG. 14) is assigned to correspond to the current location ofthe drone 105. In many preferred embodiments, the drone 105 isautomatically commanded to slow and hover in space when the transitionbetween modes is made.

In some embodiments, the transition between modes is controlledautomatically using a heuristic. In one inventive heuristic, thetransition is controlled as follows: When the drone 105 is hovering ormoving at a slow speed, the absolute position control interface (e.g. asshown in FIG. 14) is presented and used, as displayed upon the portablecomputing devices 104. But if the drone 105 is commanded to move beyondthe limits of the control area 1404 shown in FIG. 14, the mode switchesto relative control, as shown in FIG. 9. If the drone is commanded tohover, or stay at a slow speed (a speed below a threshold value) formore than a threshold amount of time (for example 10 seconds), the modeswitches back to absolute position control. In this way, the users maycollaboratively control a drone in a physical space, and collaborativelyswitch between control modes, using their collective will to choosebetween relative control mode and absolute position control modes.

Referring next to FIG. 17, in another embodiment of the presentinvention, the pointer 320 is represented by a graphical representationof the actual drone, including a camera representation 1700 of where thecamera 115 is located on the drone 105. In FIG. 17 the camerarepresentation is a triangle, where triangle location represents thelocation of the camera 115 on the drone 105.

In some such embodiments, the magnet icon embodiment (as shown in FIGS.4-7) may be used both to apply a pulling force on the location of thedrone 105, as well as a pulling force to change the orientation of thedrone 105. In this way, a group of users may collaboratively controlunder real-time swarm manipulation, both the position and orientation ofthe drone 105 through closed-loop feedback control.

Referring next to FIG. 18, a schematic diagram of an exemplarymulti-swarm collaboration system 1800 is shown. Shown are a firstCentral Collaboration Server (CCS) 1802, a second CCS 1804, a firstswarm 1806, a second swarm 1808, the plurality of computing devices 104,the drone 105, the plurality of exchanges of data 106, the dronecontroller unit 110, two server communication links 114, the camera 115,and the drone communication link 122.

In some embodiments of the present invention, the plurality of users maybe split into multiple swarms. In the example of FIG. 18, the pluralityof users is split into two swarms. The first swarm 1806 comprises aportion of the plurality of computing devices 104, each computing devicein the first swarm 1806 in communication with the first CCS 1802. Thesecond swarm 1808 comprises the remainder of the computing devices 104,which are each in communication with the second CCS 1804. Each server1802, 1804 is in turn in communication with the drone controller 110 viathe connection links 114. In other embodiments, the plurality of usersis split into more than two swarms.

In one embodiment, one swarm of users (e.g. the first swarm 1806) cancollaboratively control the motion of the drone 105, and another swarmof users (e.g. the second swarm 1808) can collaboratively control thepan, tilt, and/or zoom of the camera 115 on the drone 105. In otherembodiments of the present invention, the plurality of users may besplit into multiple swarms, with one swarm of users collaborativelycontrolling the direction of motion of the drone 105, and another swarmof users collaboratively controlling the orientation of the drone 105.In this way, swarm of users may form a plurality of intelligent systems,each of said intelligent systems controlling an aspect of the drone, theintelligent systems working together to create the full experience. Insuch embodiments, multiple CCS servers work in parallel, all connectedto the same drone controller 110.

While the above examples are generally shown with respect to flyingdrones, the current invention can be used to control other vehicles, forexample driving drones wherein the position and/or motion of the drivingdrone is fixed to ground-based operations. Similarly, water-based drones(either surface boats or underwater drones) can be controlled using themethods of the present invention. In addition, end-effectors can beadded to drones as are known the current art. These end effectors can becontrolled by user-swarms through collaborative real-time manipulation,using the multi-swarm methods disclosed above, wherein a swarm of usersare enabled to control the end effector.

While many embodiments are described herein, it is appreciated that thisinvention can have a range of variations that practice the same basicmethods and achieve the novel collaborative capabilities that have beendisclosed above. Many of the functional units described in thisspecification have been labeled as modules, in order to moreparticularly emphasize their implementation independence. For example, amodule may be implemented as a hardware circuit comprising custom VLSIcircuits or gate arrays, off-the-shelf semiconductors such as logicchips, transistors, or other discrete components. A module may also beimplemented in programmable hardware devices such as field programmablegate arrays, programmable array logic, programmable logic devices or thelike.

Modules may also be implemented in software for execution by varioustypes of processors. An identified module of executable code may, forinstance, comprise one or more physical or logical blocks of computerinstructions that may, for instance, be organized as an object,procedure, or function. Nevertheless, the executables of an identifiedmodule need not be physically located together, but may comprisedisparate instructions stored in different locations which, when joinedlogically together, comprise the module and achieve the stated purposefor the module.

Indeed, a module of executable code could be a single instruction, ormany instructions, and may even be distributed over several differentcode segments, among different programs, and across several memorydevices. Similarly, operational data may be identified and illustratedherein within modules, and may be embodied in any suitable form andorganized within any suitable type of data structure. The operationaldata may be collected as a single data set, or may be distributed overdifferent locations including over different storage devices, and mayexist, at least partially, merely as electronic signals on a system ornetwork.

While the invention herein disclosed has been described by means ofspecific embodiments, examples and applications thereof, numerousmodifications and variations could be made thereto by those skilled inthe art without departing from the scope of the invention set forth inthe claims.

What is claimed is:
 1. A real-time collaborative vehicle control systemcomprising: a plurality of computing devices each comprising acommunications infrastructure coupled to each of a processor, a memory,and a collaborative intent application stored on each memory andconfigured to run on each processor, and a user interface coupled to adisplay and configured to receive input from a user; a collaborativeserver in networked communication with the plurality of computingdevices, the collaboration server including a processor and a memory; avehicle controller configured to receive commands from the collaborativeserver; and a vehicle operatively controlled by the vehicle controller,wherein the control system is configured to perform the following steps,wherein computing device steps are performed substantiallysimultaneously on each of the plurality of computing devices: display,on each of the computing devices, vehicle control information, and acollaboratively controlled graphical indication having a coordinatelocation in relation to the vehicle control information; receive, oneach of the computing devices, repeatedly in real-time, user input of auser intent vector through the user interface, the user intent vectorhaving a direction in relation to the user control information and amagnitude, send, repeatedly in real-time, the user intent vector to thecollaboration server, responsively determine, repeatedly in real-time bythe collaboration server, a group intent vector from the plurality ofreceived user intent vectors, and determine, repeatedly in real-time bythe collaboration server, an updated coordinate location based on thegroup intent vector; determine, repeatedly in real-time by thecollaboration server, at least one vehicle command in response to theupdated coordinate location; send, repeatedly in real-time by thecollaboration server, the at least one a vehicle command to the vehiclecontroller, whereby the vehicle operates in response to the at least onevehicle command; send, repeatedly in real-time by the collaborationserver, the updated coordinate location to the plurality of computingdevices; and update, repeatedly in real-time by the computing devices,the location of the collaboratively controlled graphical indicationusing the received updated coordinate location, whereby a closed-loopsystem is formed between the collaboration server and each collaborativeintent application.
 2. The real-time collaborative vehicle controlsystem of claim 1, wherein the vehicle is a flying drone vehicle.
 3. Thereal-time collaborative vehicle control system of claim 2, wherein theat least one control axis includes at least one of a control axisindicating forward/backward motion of the vehicle and a control axisindicating leftward/rightward motion of the vehicle.
 4. The real-timecollaborative vehicle control system of claim 2, wherein thecollaboratively controlled graphical indication is influenced by asimulated spring force, said simulated spring force configured to pullthe displayed collaboratively controlled graphical indication towards anorigin position on the at least one control axis.
 5. The real-timecollaborative vehicle control system of claim 1, wherein the vehiclecontrol information includes at least one control axis, wherein alocation of the graphical indication relative to the at least onecontrol axis indicates the at least one command sent to the vehicle. 6.The real-time collaborative vehicle control system of claim 5, whereinthe first control axis and the second control axis intersect at anorigin location.
 7. The real-time collaborative vehicle control systemof claim 5, wherein the at least one control axis includes a firstcontrol axis oriented vertically and a second control axis orientedhorizontally.
 8. The real-time collaborative vehicle control system ofclaim 1, the system further comprising a camera coupled to the vehicleand operatively controlled by the vehicle controller.
 9. The real-timecollaborative vehicle control system of claim 8, wherein the camera isconfigured to stream live video to the plurality of computing devices.10. The real-time collaborative vehicle control system of claim 8,wherein each of the plurality of computing devices displays the livevideo on a same display as the collaboratively controlled graphicalindication.
 11. The real-time collaborative vehicle control system ofclaim 1, wherein the at least one vehicle command includes at least onecommand for camera operation.
 12. The real-time collaborative vehiclecontrol system of claim 1, wherein the at least one command for cameraoperation includes at least one of forward/backward tilt andleftward/rightward pan.
 13. The real-time collaborative vehicle controlsystem of claim 1, wherein the system is configured to display agraphical indication of the user input.
 14. The real-time collaborativevehicle control system of claim 13, wherein the graphical indication isa U-shaped magnet icon.
 15. The real-time collaborative vehicle controlsystem of claim 14, wherein the system is configured to change at leastone of the size, orientation, and location of the magnet icon based onthe user input.
 16. The real-time collaborative vehicle control systemof claim 1, wherein the collaboratively controlled graphical indicationis displayed on a control area corresponding to a plan area of operationof the vehicle.
 17. The real-time collaborative vehicle control systemof claim 16, wherein the collaboratively controlled graphical indicationrepresents a location of the vehicle in the control area.
 18. Thereal-time collaborative vehicle control system of claim 16, the controlarea further including a display of plan coordinate axes.
 19. Thereal-time collaborative vehicle control system of claim 1, wherein thecollaboratively controlled graphical indication is an icon indicatingthe orientation of the vehicle.
 20. The real-time collaborative vehiclecontrol system of claim 1, wherein the collaboratively controlledgraphical indication has at least one of a simulated mass value and asimulated damping value, the at least one of the simulated mass valueand the simulated damping value selected to slow a displayedcollaboratively controlled motion of the collaboratively controlledgraphical indication to better match dynamics of the vehicle.